Statistical Significance in Inductive Learning

Inductive learning systems search for regularities that therefore be applied with some assurance to an example describe environmental observations, These systems often use which does not belong to the learning set. In other numeri~~l heu~stics to guide this search, The~ also sele~t words, statistical significance may be used to assess regulantles which are good, or the best, according to certain that a learnin g Program reall y learned th ' d d ' d , 1 ' , S " , al h 1 " d some mg an 1 numenca cntena, tatlstlc measures ave recent y game t I ' d t f t 1 " , th AI ' Th ' " d " 1 no on y provl e ar e ac s, popu anty m e communIty, IS paper provi es a simp e method for fully exploiting statistical measures, We give a test However, there is a difficulty. Let us continue our which may be used to decide whether a given regularity is previous example. A classification rule provided by a statistically significant, or, in other words, whether this regularity system is found by optimizing a given measure in the may be distinguished from random" This method directly uses learning set. Therefore, the score obtained in the results obta~ed ~rom the learning set with~ut ~qui,ring any test learning set by this rule with this measure, inevitably set. An application to Concep~al Oustenng IS given and ,the provides an optimistically biased view of the real performance of the method IS evaluated by a numencal " " , ul ' P" all ' d f 1 d d ' d performance of the rule, The same holds when CrIterIa slm atlon, In y, we provi e Ie erences to Ie ate stu les an, , d b tabl t " wlth related semantIcs are used, e. g. entropic and some e a e ques Ions. likelihood ratio criteria in CN2. Moreover, because of the (large) number of possible rules, there are always